The present invention relates to the field of testing and more particularly that of testcase management.
It is vital to ensure that a product or system is fully operational and consistently performs according to its functional specification before it is made available to the public. The reliability of computer hardware/software is especially important since computers form the backbone of an increasingly large number of organisations. When a computer system fails to respond as intended, businesses are invariably unable to provide even the most basic of services. Money, reputation or even lives may be lost, dependant upon the criticality of the service, the outage time etc.
In today""s increasingly competitive market-place, quality and reliability are of the utmost importance. Customers do not tolerate mistakes and the later a defect is discovered, the more costly it can prove to the manufacturer. Exhaustive testing is impractical, if not impossible, but what is important however is that a computer system is subjected to as many operational scenarios as is feasible. Any resulting problems can then be corrected before the system is released.
Typically, these operational scenarios may be simulated using a large number of small computer programs known as testcases. Each testcase, within an overall test suite, is designed to test a different aspect of the system. A test harness is used to run the suite of testcases as well as performing other tasks such as finding testcases in a directory tree and producing a report describing which testcases passed and which ones failed.
A test run may comprise a thousand or more testcases and if the testcases themselves cannot be relied upon, all failures have to be analyzed to determine whether or not the reported fault actually exists. This investigation greatly increases the time taken before the true defects in the system under test get reported to the developers and ultimately before a fix can be engineered.
During the test phase of a computer software system, it is quite usual for the test suite itself to be in a dynamic state of change. New testcases will be created, whilst old ones are altered. The writing and testing stages thus typically operate in parallel with one another.
This approach is particularly advantageous since it means that the testcase writers no longer have to assimilate full knowledge about all aspects of the computer system before creating the whole suite of cases. Instead they may apply the experience that they gain with time and use the feedback received from the testers themselves to ensure the thoroughness and applicability of their work.
Another advantage, is that testcases may be written by a variety of different people with varying amounts of time and ability to devote. By running the two phases in parallel, it is possible to start with a small selection of tests and to add to them as and when the relevant personnel become available. Thus time is not wasted waiting for a testcase writer to finish another project or return from sick leave, holiday etc.
Furthermore, during the development of a product, it is extremely likely that a customer""s requirements will alter somewhat. It is important to be able to adapt to the resulting specification changes during the test phase. This too is made a great deal easier by running both phases simultaneously.
All of the above should culminate in a much more efficient modus operandi. Ultimately, the time taken to get the system to the market-place should be greatly reduced. The customers are kept happy and the system can start recouping the investment placed in it.
Despite the merits of a continually changing and adapting suite of testcases, there are however a number of associated problems:
i) New testcases often contain defects and as a result, they may omit to detect a current flaw in the aspect of the system under test (false positive). Alternatively, the testcase may report a non-existent fault (false negative).
ii) Some testcases are interactive, providing the tester with a variety of instructions. These may ask the tester to perform a particular operation and also list the appropriate pass/fail criteria. For example, the tester may be told to activate a popup menu and informed that this should contain three menu-items. Both the task instructions and the pass/fail criteria are frequently incomplete or difficult to interpret. This is usually because the testcase writer does not know who will run the testcase or the amount of knowledge that person has. Furthermore, an instruction that is obvious to the testcase writer may not be so easy for others to decipher.
iii) Further problems can occur if the environment under which the testcase was developed differs from that in which the testcase is being run. A testcase created under a particular operating system, for instance, may rely upon a feature of that operating system which is not universally available. An example of this is the xe2x80x9cvmstatxe2x80x9d command which reports virtual memory statistics on UNIX based systems, but has no meaning elsewhere. A testcase incorporating such a command would in all likelihood fail if executed on, for example, a Windows NT system. Furthermore, certain testcases are required not to be run when a feature they test or use is not installed or activated.
iv) A testcase may also fail if the environment in which it is executed is not configured in a particular way. It may rely upon certain resource restrictions and require, for example, that the amount of free memory is below a pre-determined level. Any other memory setting may cause the testcase to abnormally end.
v) Furthermore a testcase may provoke a failure in the test harness. A failure in the harness itself will prevent the bona fide testcases from being run and delay the test phase.
Such failures cause testers many hours of painstaking investigative work and can severely hold up proceedings. It is important for testers to be able to rely upon the testcases themselves and to be sure that any failures are as a result of true defects in the system under test.
There are a number of additional reasons for not running a particular testcase:
i) During the development lifecycle, it is quite common to defer the fixing of a defect in the system under test until a future version. Testing for that defect in an earlier version will naturally also cause the associated testcase to fail. Testers testing the current version of the system do not want to keep running testcases which show up these deferred defects.
ii) It is also common for different teams to be testing different versions of the system at the same time. These teams will want to run different versions of the test suite which contain a set of tests appropriate to the version they are testing.
iii) During the development of a system, at any one time, a number of system defects may have been reported to the development team but have not yet been fixed. Information regarding each defect is stored as a defect record. Each record contains a status field which will have a value of xe2x80x98openxe2x80x99, xe2x80x98workingxe2x80x99, xe2x80x98verifyxe2x80x99 or xe2x80x98closedxe2x80x99. xe2x80x98Openxe2x80x99 means that the defect has been reported to development but they have not yet started work on it. xe2x80x98Workingxe2x80x99 means that the development team are working on the defect. xe2x80x98Verifyxe2x80x99 means that development have applied a fix for the defect and they require the test team to verify the fix is good by re-running the testcase. xe2x80x98Closedxe2x80x99 means that the test team have verified that the fix is good. System defects will move between the four states whilst the development activity is in progress. Testcases are re-run on a regular basis and there is little point in re-running testcases which provoke defects whose status is xe2x80x98openxe2x80x99 or xe2x80x98workingxe2x80x99 because known failures would keep re-occurring and this would tend to swamp the new failures.
Testers do not want to be concerned with those testcases which will fail for any reason, other than that there is a real problem with the system under test. It is vital, therefore, for them to be able to effectively manage the testing phase by executing only those testcases which are applicable to the current set of circumstances. They must be able to exclude any rogue/problematic testcases.
A number of attempts have been made to enable this. It is possible to avoid such failures by unhooking the bona fide testcase(s) from the suite and running them in an isolated fashion. This however is a cumbersome and time-consuming approach. Furthermore, the tester has to be aware of the current status for all testcases in order to make an informed decision. Since suites will typically be added to in an ad hoc fashion by a number of different people, this is just not feasible.
Another alternative is to categorise the testcases by inserting a keyword or phrase into each. The tester is then able to instruct the test harness to run only those tests which contain keywords from a given list. For example, a set of testcases could include the keyword xe2x80x9cRunablexe2x80x9d, whilst another set includes the key phrase xe2x80x9cSystem Defect not fixedxe2x80x9d. Such a procedure however requires modification to the testcase itself and thus there is scope for the introduction of errors.
Another approach requires the maintenance of several test suites, one for each system version/feature set. Testcases found to be defective or those relating to defects not fixed/fixed in a later version are disabled. This has the disadvantage that many testcases will be present in more than one test suite and this will consume more disk space. Additionally, if a modification is made to a testcase then all the copies in the other test suites must be located and updated. This consumes a great deal of time and is very prone to error. This method also requires the maintenance of a list of disabled testcases so that the testcase writers know which ones need fixing.
Configuration Management Version Control product (CMVC) from the IBM Corporation is a system which is used to control changes to any software system which is made up from a large number of source files. It allows the tracking of defects in the system being developed as well as any proposed changes to the system. It permits any updates or modifications to be properly controlled and allows these to be undone should their effect be undesirable. CMVC is intended to manage a large number of inter-related files to build a software system. It has no features useful to someone who wants to effectively manage the exclusion of defective or unapplicable testcases. However, CMVC files are a useful way to store testcases because of the automatic backup and early version recall features.
Test Tracking Tool (TTT) from the IBM Corporation comprises a set of Lotus Notes databases which enable the management of data associated with the testing phase of a project. The tool aids testcase definition, test preparation and test execution. Multiple testcases may be assigned a status of xe2x80x9cblockedxe2x80x9d and thus excluded during the test phase. However, such a method of excluding testcases only operates within the TTT environment. It is not portable to other systems.
It is possible to categorise the testcases using a library system. A plurality of folders may be created with a meaningful heading assigned to each. Such headings could include: xe2x80x9cfaulty testcasesxe2x80x9d, xe2x80x9csystem defect fixed in version 1.1xe2x80x9d, xe2x80x9csystem defect not fixedxe2x80x9d, xe2x80x9cTunable testcasesxe2x80x9d etc. Each testcase is then placed in the relevant directory, with certain directories being excluded during the test run according to the system under test.
During the test phase however, testcases will need to swap directories as and when their status changes. For example, once a bug has been fixed in testcase A, that testcase will have to be moved from the xe2x80x9cfaulty testcasesxe2x80x9d category to the xe2x80x9crunable testcasesxe2x80x9d category. Testcases will therefore be continually on the move. When a suite of testcases is executed, the test harness has to be provided with a path indicating where the testcases themselves sit. With a library system of categorisation, this path would be continually changing and this would prove extremely confusing for the testers.
Batch Environment Random Driver 1.0 (BERD) from the IBM Corporation is a workload generator which creates a batch workload by randomly or sequentially submitting jobs from user specified libraries at a user specified rate. Tying this into the present context, batch jobs can be thought of as testcases. The tool also supports an exclusion list which enables the identification of jobs or testcases which are not to be submitted. This is an improvement on the library system since the testcases do not need to move folders according to their current status. Instead, a single entry is made on, or removed from the exclusion list. This is a much simpler way of managing testcases. However, there may be multiple reasons for excluding a testcase as explained above and testers need to be able to track this effectively. BERD does not provide this additional level of functionality/flexibility.
According to the preferred embodiment, multiple lists defining different categories of testcases which may be excluded during a test run and a program to stop those testcases in the lists from running are utilised. Different testers can choose to exclude different tests, causing different combinations of the lists to form an overall list, thus including only the tests appropriate to the system being tested. For this reason, more than one copy of the overall list may exist, with each copy being appropriate to a different set of circumstances. It should be seen that the overall list can either be a list of cases which are to be excluded or it may be the converse including only cases which are to be used in a test run.
The multiple lists can also be used to ensure that only testcases which have been fully tested in all environments are run. This means that time is not wasted by testers investigating problems generated by rogue testcases or testcases with inadequate instructions. Testers are periodically notified if one of their testcases is rogue and they can then repair it. This notification may be manual, by e-mail or some other means. Testcases which have not been tested may also be excluded until verified. A testcase may be excluded for more than one reason. The multiple lists can also be used to mitigate the effects of faults in a testcase harness.
Furthermore, the multiple lists can be used to control which testcases are run depending on the development state of the system under test. For example, there is little point in executing testcases relating to known system defects since these will fail. The multiple lists are also used to control the development of testcases and prevent testers wasting time running testcases for system defects which have not yet been fixed or which have been fixed in a later version of the system under test.
System defects may only have been fixed in the version of the system under test for a particular operating system. It is also possible using the invention to ensure that testcases relating to these system defects are not run.
Multiple lists allow testers to effectively categorise the testcases. The testcases themselves are left untouched meaning that inadvertent errors are not introduced. Whenever a testcase changes category (eg moves from a xe2x80x9cfaulty testcasesxe2x80x9d category to a xe2x80x9crunable testcasesxe2x80x9d category) only an entry on one of the lists needs be changed. There is no longer any need to maintain a different test suite for each environment/set of circumstances. Rather the lists identify which testcases within the overall suite to run. Testcases remain in one place and thus the path that must be provided to the testcase harness to tell it where to find its testcases does not keep changing. In this way, confusion is avoided. This method of excluding testcases is also portable. It does not rely upon a specific testing environment.